Some computing systems include multiple components which interact, particularly in which those computing systems are relatively complex and in which there is substantial code directing their operation. It might occur that system coders or system designers have devised one or more “test suites”. Test suites are disposed for emulating possible conditions which the system might undergo, and for determining if code for handling those possible conditions handles those conditions correctly. For just one example, in some complex systems there might occur conditions in which a 1st element locks a data structure, while a 2nd element attempts to access that data structure. If the lock is not performed properly, the 2nd element might be able to alter that data structure at times in which the 1st element relies on its being unchanged, which could result in an error.
A 1st problem in the known art occurs in cases in which the multiple components are situated at, or controlled from, relatively distributed elements. Such relatively distributed elements might include those which are logically or physically relatively disparate or remote. This can present a problem when it is desired that the test suite coordinate actions of those relatively distributed elements, such as for example, causing a 1st such element and a 2nd such element to operate in conjunction, or to operate in relative time proximity. For just one example, in some systems there might occur conditions in which a 1st element performs two operations A and B in relatively quick succession, while a 2nd element performs an operation X between those two. Thus, while the 1st element expects (A then B), the actual sequence is (A then X then B). If the system is not well-designed, or if the code is not well-disposed for handling this case, this could result in an error.
Known systems include designing test suites so that the 1st element and the 2nd element attempt to coordinate their operations in time succession. While these known systems might be able to adequately test some cases, they have at least the drawback that there appear to be cases which these known systems are unable to test adequately and reliably.
A 2nd problem in the known art occurs in cases in which the test suite requires for operation a particular set of components or component types. For just one example, testing a locking mechanism in a distributed system would likely involve a data structure to be locked, a component A that locks that data structure, and a component B that attempts to access that data structure despite the lock. In some cases, it might be desirable for the system to reject the attempted access to the locked data structure; in other cases, it might be desirable for the system to break the lock, allow the attempted access, and so inform the component A that set the lock. In such cases, it would generally be required to identify components A and B, and the lockable data structure, before proceeding with the test suite.
Known systems include operating test suites “by hand”, that is, using a human operator to select the components to be included in the test suite, using a human operator (possibly the same one) to cause the test suite to operate on those components, and using a human operator (again, possibly the same one) to review a set of results from the test suite. One such known system includes pre-allocating multiple devices which each attempt to access and modify a locked data structure. Once the operator has selected the multiple devices and caused each of the multiple devices to run the test suite, the operator can check to see if the data structure includes a data value which would occur if the system were operating correctly. While this known system might be able to adequately test some cases, it has at least the drawback that it involves significant operator intervention just to perform the test.
A 3rd problem in the known art occurs in cases in which there is more than one such test suite to perform. In some cases, the more than one test suite might re-use the same resources. In some cases, the more than one test suite might use distinct resources, but an order of allocation of those resources might have a substantial effect on the efficiency, or another measure of performance, of the test suite.